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, Abstract 

^ ■ Wc consider a class of dynamic advertising problems under uncertainty in the presence 

of carryover and distributed forgetting effects, generalizing the classical model of Nerlove 
and Arrow |32| . In particular, we allow the dynamics of the product goodwill to depend on 
its past values, as well as previous advertising levels. Building on previous work ([16]), the 
optimal advertising model is formulated as an infinite dimensional stochastic control prob- 
' lem. We obtain (partial) regularity as well as approximation results for the corresponding 

■ value function. Under specific structural assumptions we study the effects of delays on the 

value function and optimal strategy. In the absence of carryover effects, since the value 
function and the optimal advertising policy can be characterized in terms of the solution 
of the associated HJB equation, we obtain sharper characterizations of the optimal policy. 
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^ ■ 1 Introduction 
in 

This paper is devoted to the study of a class of optimal control problems for linear stochastic 
differential equations with delay both in the state and the control term, and is a natural con- 
. tinuation of [16] . These problems arise in the theory of optimal advertising under uncertainty 

with memory structures. We approach the problem using stochastic control techniques in 
infinite dimensions. 

In particular, in [16] we considered a controlled stochastic differential equation (SDE) 
with delay entering both the state and the control variable as an extension of the dynamic 
advertising model of Nerlove and Arrow [32] . The results of [16] are the following: we construct 
^ ■ a controlled infinite dimensional SDE that is equivalent to the controlled SDE with delay, we 

prove a verification theorem, and we exhibit a simple example for which the Bellman equation 
associated to the control problem admits a sufficiently regular solution, hence the verification 
theorem can be applied. In the present manuscript, we extend [16] by developing several new 
sets of results. On the one hand, we provide qualitative characterization of the first- and 
second-order properties of the optimal value function (Section Ell). In particular, we show 
that, under natural restrictions, the monotonicity of the optimal value function with respect 
to the initial goodwill profile still holds even in the presence of the state and control-related 
delay terms in the advertising dynamics (Proposition 13.61) . In addition, we establish that the 
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decreasing marginal influence of the attained goodwill levels on the primal profit components is 
retained by the optimal profit function (Proposition l3.2|) . As is well known, this last property is 
important in reducing the computational load required to solve the time- and space-discretized 
version of our problem by dynamic programming methods. On the other hand, in the view of 
intractability of the general variant of our problem, we propose approximation schemes for the 
optimal value function (Theorem l3.7|) and for the optimal advertising policy (Propositions 3.11 
and 3.12). The latter result is of particular importance since it suggests a computationally 
feasible approach to constructing asymptotically optimal advertising trajectories. In addition, 
we provide a complete characterization of the optimal advertising policy in the case when the 
cost function is quadratic and the reward function is linear in goodwill level (Section 3.3). 
For a specific instance of this case we conduct a numerical study aimed at demonstrating the 
importance of proper accounting of the delay effects in calculating the optimal advertising 
policy. 

Finally, we are able to provide sharper characterization of the optimal policies in the case 
when the influence of advertising on the goodwill evolution is instantaneous and the delay 
effects are of the state-only type (Section |3]). The key result in this section is Theorem 14.51 
which formulates sufficient conditions ensuring that the optimal advertising policy is of the 
feedback type. In particular, in the case of linear cost function the optimal control takes a 
particularly simple "bang-bang" form (Corollary 14. 10|) . 

Optimal control problems for stochastic systems with delay in the state term admit alter- 
native, more traditional treatments: for instance, see [TO] and [25] for a more direct application 
of the dynamic programming principle without appealing to infinite dimensional analysis, and 
|23| for the linear-quadratic case. However, we would like to point out that none of the meth- 
ods just mentioned apply to the control of stochastic differential equations with delay in the 
control term. 

Analysis of advertising policies has always been occupying a front-and-center place in the 
marketing research. The sheer size of the advertising market (over $143 billion in the US 
in 2005 |31| ) and the strong body of evidence of systematic over-advertising by firms across 
many industries (see e.g. [T], [20], [29], [30]) has caused a renewal of attention to the proper 
accounting for the so-called "carryover" or "distributed lag" advertising effects. The term 
"carryover" designates an empirically observed advertising feature under which the advertising 
influence on product sales or goodwill level is not immediate, but rather is spread over some 
period of time: according to a survey of recent empirical "carryover" research by Leone [28] . 
delayed advertising effects can last between 6 and 9 months in different settings. 

On the theoretical front, pioneering work of [33] and [35] has paved the way for the devel- 
opment of a number of models dealing with the optimal distribution of advertising spending 
over time in both monopolistic and competitive settings. A comprehensive review of the state 
of the advertising control literature in [11] points out that the majority of these models operate 
under deterministic assumptions and do not capture some of the most essential characteristics 
of real-world advertising phenomena. On the empirical side, one of the first and most im- 
portant substreams of advertising literature was formed by the papers focused on the studies 
of distributed advertising lag (see e.g. [3], [5], p2]). An important early empirical result 
was obtained by Bass and Clark [3], who established that the initially adopted models with 
monotone decreasing lags (see [23]) are often inferior in their explanatory power to the models 
with more general lag distributions. 

Despite the wide and growing empirical literature on the measurement of carryover effects, 
there are practically no analytical studies that incorporate distributed lag structure into the 
optimal advertising modeling framework in the stochastic setting. The only papers dealing 
with optimal dynamic advertising with distributed lags we are aware of are [5] (which provides 
a numerical solution to a discrete-time deterministic example) , and [TO] , [TO] (which applies a 
version of the maximum principle in the deterministic setting) . The creation of models which 
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incorporate the treatment of "carryover" effects in the stochastic settings have long been 
advocated in the advertising modeling literature (see e.g. [TH], [llj and references therein). 

As mentioned above, in this work we study a class of stochastic models deriving from 
that of Nerlove and Arrow [32] , incorporating both the advertising lags as well as distributed 
"churn" ([33]), or "forgetting", effects. More precisely, we formulate an optimization program 
that seeks to maximize the goodwill level at a given time T > net of the cumulative cost 
of advertising until T. This optimization problem is studied using techniques of stochastic 
optimal control in infinite dimensions, using the modeling approach of [16]: in particular, we 
specify the goodwill dynamics in terms of a controlled stochastic delay differential equation 
(SDDE), that can be rewritten as a stochastic differential equation (without delay) in a suitable 
Hilbert space. This allows us to associate to the original control problem for the SDDE an 
equivalent infinite dimensional control problem for the "lifted" stochastic equation. 

The paper is organized as follows: in section [2] we formulate the optimal advertising prob- 
lem as an optimal control problem for an SDE with delay, and we recall the equivalence result 
of [16] . In section Owe prove the above mentioned results about the value function and ap- 
proximate strategies in the general case, together with a detailed discussion of the effect of 
delays in a specific situation. Section U] treats the case of distributed forgetting in the absence 
of advertising carryover. 

Let us conclude this introduction fixing notation and recalling some notions that will be 
needed. Given a lower semicontinuous convex function /:£-»t:=MU {+00} on a Hilbert 
space E with inner product (•, •), we denote its conjugate by f*(y) := sup x£E ({x, y) — f(x)). 
Recall also that D~ f*(y) = argmax xg £;((x, y) — f(x)), where D~ stands for the sub differential 
operator (see e.g. [2], p. 103). Throughout the paper, X will be the Hilbert space defined as 

I = Rx L 2 ([-r, 0],R), 

with inner product 

,0 

(x,y)=x y + sci(f)yi(f)d£ 

J — r 

and norm 

M = (Vo| 2 + J M£)l 2 ^ 

where r > 0, xq and xi(-) denote the M-valued and the L 2 ([— r, 0], M)-valued components, 
respectively, of the generic element x of X. Given / : X — > R, k £ {0, 1}, we shall denote 
by dkf, DZ f, respectively, the partial derivative and the subdifferential of / with respect to 
the k-th component. We shall use mollifiers in a standard way: for £ £ C°°(R d ,R+), equal 
to zero for \x\ > 1 and such that J Rd £(x) dx = 1, we shall set (\(x) = X~ d ((X~ 1 x) for A ^ 0. 
By a < b we mean that there exists a constant iV such that a < Nb. If N depends on some 
parameter of interest p, we shall write N(p) and a < p b. 

2 The model 

We consider a monopolistic firm preparing the market introduction of a new product at some 
time T in the future. In defining the state descriptor for a firm to follow we use the Nerlove- 
Arrow framework and consider the product's "goodwill stock" y(t), < s < t < T. The firm 
directly influences the rate of advertising spending z(t) to induce the following trajectory for 
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the goodwill stock: 



ai(Ov(t + 0^ + b z(t) + f h(0<t + d£ 

J —r 



dy{t) = a y(t) + 

+adW (t), s<t<T 
{ y(s) = x ; y(s + £) = MO, z(s + = $(0, Z G [~r, 0], 



dt 



(1) 



where the Brownian motion Wo is defined on a filtered probability space (fi, F = (J r t)j>o, IP), 
with F being the completion of the filtration generated by Wo. We assume that the advertising 
spending rate z(t) is constrained to remain in the set U, the space of F-adapted processes taking 
values in a compact interval U C R + . In addition, we assume that the following conditions 
hold: 

(i) a < 0; 

(ii) ai(-)£i 2 ([-r,0],R); 

(iii) b > 0; 

(iv) M')ei 2 (hO],R+); 

(v) x > 0; 

(vi) xi(-) > 0, with xi(0) = xo; 

(vii) $(■) > 0. 

Here ao and ai(-) describe the process of goodwill deterioration when the advertising stops, and 
bo and &i(-) provide the characterization of the effect of the current and the past advertising 
rates on the goodwill level. The values of xo, xi(-) and 5(-) reflect the "initial" goodwill and 
advertising trajectories. Note that we recover the model of Nerlove and Arrow from (fTJ) in the 
deterministic setting (a = 0) in the absence of delay effects (ai(-) = bi(-) = 0). 

Setting X 3 x := (xo, x\(-)) and denoting by y s ' x ' z (t), t G [0, T], a solution of ([T]), we define 
the objective functional 

rT 



J(s, x; z) = E 



(2) 



where ipo '■ M — ► R and /io 
satisfying the conditions 

and 



My s ' x ' z (T))- J h (z(t))dt 

_ — > M + are measurable utility and cost functions, respectively, 
¥o{x)\ < K(l + \x\) m VxeM, (3) 



|/i(x)|<i<: Vxet/, (4) 

for some K > and m > 0. In the sequel we shall often move the superscripts s, x, z to the 
expectation sign, with obvious meaning of the notation. Let us also define the value function 
V for this problem as follows: 

V(s, x) = sup J(s, x; z). 
zeu 

We shall say that z* S IA is an optimal strategy if it is such that 

V(s, x) = J(s, x; z*). 

The problems we will deal with are the maximization of the objective functional J over all 
admissible strategies U, and the characterization of the value function V and of the optimal 
strategy z*. 

Throughout the paper we will always assume that the assumptions of this section hold 
true. In particular the constants T, m and K are fixed from now on. 
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2.1 An equivalent infinite dimensional Markovian representation 

We shall recall a representation result (proposition 12.11 below) proved in [16], generalizing a 
corresponding deterministic result due to Vinter and Kwong [35]. 
Let us define an operator A : D(A) Cl-»las follows: 

A : (x ,xi(f)) i ^ (a x + gi(0)> Qi(C) g o ~ ) a.e. f G [-r, 0] , 

D(^) = {x G X : xi G W 1,2 ([-r, 0] ; R) , xi(-r) = 0} . 
Moreover, define the bounded linear control operator B : U — > X as 

B : u i— > ( bQU, bi(-)u) , (5) 



and finally the operator G : R — ► X as G : xq i— > (<rxo,0). Sometimes it will be useful to 
identify the operator with the element (6q, b\) G X. 



(6) 



Proposition 2.1. Ze£ Y(-) &e i/ie weak solution of the abstract evolution equation 

( dY(t) = (AY(t) + Bz(t))dt + GdW {t) 
\ Y(s) = i£l, 

with arbitrary initial datum x G X and control z G IA. Then, for t > r, one has, P-o.s., 

Y(t)=M(Y (t),Y (t + -),z(t + -)), 

where 

M : X x L 2 ([—r, 0], R) -» X 

(x ,xi(-),f(-)) ^ Oo,m(-)), 

m(0 := y ? a 1 (C)x 1 (C-C)dC + ^ MCMC - <*C- 

Moreover, let {y(t), t > —r} be a continuous solution of the stochastic delay differential 
equation |7p, and Y(-) 6e i/ie weafc solution of the abstract evolution equation {6]) with initial 
condition 

x = M(x ,xi,<5(-)). 

Then, for t > 0, one has, F-a.s., 

Y(t)=M(y(t),y(t + -),z(t + -)), 
hence y(t) = Yo(t), F-a.s., for all t > 0. 

Using this equivalence result, we can now give a Markovian reformulation on the Hilbert 
space X of the problem of maximizing ([2]), as in [16]. In particular, denoting by Y s ' x ' z (-) a 
mild solution of ([6]), ([2]) is equivalent to 



J(s, x; z) = E 



^(y s ^(r))+ / h{z(t))dt 



T 



(7) 



with the functions h : U — > R and : X — ► R defined by 

h(z) = -h (z) 
tp(x ,xi) = ip {x ). 

Hence also V(s, x) = sup^ gW J(s, x; z). 
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3 The case of delay in the state and the control term 



The aims of this section are the following: to prove regularity properties of the value function, 
to develop an approximation scheme for the value function and the optimal strategy, and to 
illustrate in a numerical example the effects of the delay structures in our model. In particular, 
we prove that, under natural assumptions, the value function is continuous in both arguments, 
and monotone concave with respect to the initial goodwill profile. As already remarked, this 
property is essential in order to obtain computationally tractable discrete-time and discrete- 
state-space dynamic programming versions of our problem. 

Moreover, since we cannot guarantee that the Bellman equation associated to our control 
problem admits a solution in general (nor do we have any information about its uniqueness 
and regularity) , it is of primary interest to obtain approximation schemes for the optimal value 
function and for the optimal advertising policy. The latter result is of particular importance 
since it suggests a computationally feasible approach to constructing asymptotically optimal 
advertising trajectories. 

In addition, in the last subsection we provide a complete characterization of the optimal 
advertising policy in the case when the cost function is quadratic and the reward function is 
linear in goodwill level, and for a specific instance of this case we conduct a numerical study 
aimed at demonstrating the importance of proper accounting of the delay effects in calculating 
the optimal advertising policy. 



3.1 Qualitative properties of the value function 

Let us first show that the value function is finite. 

Proposition 3.1. There exists a constant N = N(T,m, K) such that \V{s,x)\ < iV(l + |x|) m 
for all s € [0,T], x£X. 

Proof. The estimate from below simply follows by taking a constant deterministic control. For 
the estimate from above we have, recalling that h(x) < for all x 6 U, 



V(s,x) < supE* 



h(z(t))dt + \<p(Y(T))\ 



< K S upEl x (l + \Y(T)\Y 
zeu 



Moreover, we have 



E\Y(T)\ m < \e {T - s)A x\ m + E 



T- 



e {T - s - r)A Bz{r) dr 



+ E 



T- 



D (T-s-r 



)A GdW{i 



< T Mp\x\ m + \B\ m M^z m + E\W A (T)\ m , 



where z := max{z : z € U}, Wa($) '■= f S ^ s r ^ A G dW(r), and My := sup tg [ T ] < 
Recalling that GG* is of trace class, hence 

r-T-S 



I A 



E\W A (T)\ 2 = Tr 



e tA GG*e tA * dt < oo 



(see e.g. [7], Proposition 2.2), i.e. Wa{T) is a well-defined Gaussian random variable on X, 
we also get that E|H / A(r)| m < oo. The proof is completed observing that the upper bound 
on E| Y(T) | m is uniform over z. □ 

We establish now some qualitative properties of the value function that do not require 
studying an associated Bellman equation. The following simple result, typical of control 
problems with linear dynamics, asserts that the value function inherits the concavity with 
respect to the space variable from the reward and cost functions. 
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Proposition 3.2. If ip and h are concave, then the value function V(s,x) is proper concave 
with respect to x. 

Proof. Properness follows by the previous proposition. Moreover, let x 1 , x 2 £ X. Then 

Wis^ 1 ) + (1 - X)V(s,x 2 ) = AsupEf f h(z(t))dt + <p(y s > xl > z (T)) 

+ (l-A)supE / h(z(t))dt + ip(y s > x2 > z (T)) 
zeu Us 

= sup E ( [Xh{z 1 {t)) + {l-X)h{z 2 (t))]dt 
z\2 2 ew Us 

+ \if(y s ' xl ' zl (T)) + (1 - XMy s ^^(T)) 

Since hi is a convex set and h is concave, then z\ := Xz 1 + (1 — X)z 2 is admissible for any 
choice of z , z 2 € U, and one has 



h{z x {s)) > Xh{z l {s)) + (1 - X)h{z 2 {s)). 
Moreover, by linearity of the state equation, it is easy to prove that 

y s,x x ,z x ^ = X y s ^^(T) + (1 - X)y s ' x2 ' z2 (T), 
hence, by the concavity of (p, 

ip(y s ^(T)) > Xip(y s,xl ,zl (T)) + (1 - X)^ x ' '-* a \T)) . 
Therefore, as a consequence of (|8|) and ([9]), we obtain 



(8) 



(9) 



XV(s,x 1 ) + (l-X)V{s,x 2 



< sup E 

z x &A 



h(z x (t))dt + ^(y s > x ^(T)) 



< supE 



T 



h{z(t))dt + <p(y s ^ z (T)) 
Vis^x 1 + (1- A)x 2 ), 



which proves the claim. 



□ 



As a consequence of the previous propositions we obtain the following regularity result. 
Of course it would be ideal to obtain a result guaranteeing that V E C 0,1 ([0, T] x X), so that 
a verification theorem could be proved. Unfortunately we have not been able to obtain such 
result. We shall prove though that V is locally Lipschitz continuous in the X-valued variable. 

Corollary 3.3. Under the hypotheses of Proposition the value function V(s,x) is locally 
Lipschitz continuous with respect to x & X. Moreover, the subgradient dV(s,x) with respect 
to x exists for all x € X and is locally bounded. 

Proof. The first assertion comes from the fact that a concave locally bounded function is 
continuous in the interior of its effective domain (see e.g. Theorem 2.1.3 in [2]) and V(s,x) 
is finite for all x £ X. Corollary 2.4 in [9] and the fact that D{(j})° C D(d(p) for any concave 
function 4>, where A° denotes the interior of a set A, imply that V(s, x) is locally Lipschitz in 
x, so the assertion on dV follows. □ 
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Since V(s,x) = V(s, xq, xi) is a concave function of xq for fixed s and x±, one can also 
say that V is twice differentiable almost everywhere with respect to xq, as it follows by the 
Busemann-Feller theorem. A similar statement is not true regarding differentiability with 
respect to x%, as the Alexandrov theorem is in general no longer true in infinite dimensions. 
We now prove that the value function is continuous with respect to the time variable. It 
is possible to prove local Lipschitz continuity of V(s, ■) without appealing to concavity, but 
assuming local Lipschitz continuity of ip. 

Proposition 3.4. The value function V(s,x) is continuous in s. Moreover, if \(fio( x ) ~ 
fo(y)\ < K{\ + R) m \x — y\ for all \x\, \y\ < R, then the function V{s,x) is locally Lipschitz 
continuous with respect to x. Furthermore, there exists a constant N = N(K,m) such that 

\V(s,x) - V(s,y)\ < N(l + R) m \x - y\ (10) 

for all \x\, \y\ < R. 

Proof. Recalling that the difference of two suprema is less or equal to the supremum of the 
difference, we have 

\V(s,x) - V(s,y)\ < sup | J(s, x; z) - J(s,y;z) \ 

z&A 

< 8upR\<p(Y z > 8 < x ) - cp(Y z ' s ' y )\, 

z&A 

and, by Cauchy-Schwarz' inequality, 

E|^(yw( T )) _ y,(yw(r))| < 

KfE(l + \Y z ' s < x (T)\ m + \ Y z ' s ' y (T)\ m ) 2 ) 1/2 (e\Y z ' s ' x (T) - Y z ' s ' y {T)\ 2 Y /2 . 



Arguing as in the proof of Proposition 13. H there exists a constant N± = N\ (K, m) such that 
E\Y z ' s ' x (T)\ m < Ni(l + \x\ m ). Furthermore, a simple calculation reveals that \Y z > s > x {t) - 
Y z ' s <y(t)\ = | e ('- s ) A (x - y)\, hence 

E \Y*>* X (T) - Y Z ^(T)\ 2 < Ml\x - y\ 2 , 

and the second claim is proved. Let us now prove that V(s,x) is continuous in s for a fixed 
x. Let s n | s be a given sequence (the case s n J, s is completely similar). Bellman's principle 
yields 

V(s n ,x) = supE I h(z(r))dr + V(s,Y z > s "> x (s)) , (11) 

z&A \_J s n 

and choosing a 1/n-optimal strategy z n in ([PI]) , we have 

limsupV(s n ,x) — V(s,x) < limsupE / h{z n (r))dr 

n^oo rwoo J Sn ^2) 

+ limsupE|y(s,y^' s "' x (s)) - V(s,x)\. 

n— >oo 

The first term on the right-hand side in (| 12f) is zero because h is bounded on U. Let us show 
the also the second term on the right-hand side of (|12|) is zero: in fact we have 



, rs rs-s n 

E\Y Zn ' Sn ' x (s) -x\ 2 < K[E \e^ s - r ^ A h{z n (r))\ 2 dr + E / e^'^-^G dW( 

^ Js n JO 

< Ki(s - s n ) + q n -> 



S 



as n — ► oo, hecause h is bounded on U and the stochastic convolution is a Gaussian random 
variable with covariance operator going to as n — > oo. Therefore we also have y 2 «' s n x (s) — > x 
in probability. By (I10p . V(s,x) is continuous in x uniformly with respect to s, hence 

V( s ,Y z "' s "' x (s)) -> F(s,x) 

in probability by the continuous mapping theorem. Moreover, recalling that \V(s, x)| < N(l + 
\x\ m ) and Esup 4g [ 0j r] |y(t)| m < oo, we have 

E\V(s, Y z "' Sn ' x (s)) - V(s, x)\ m < oo, (13) 

hence Vitali's theorem implies 

E\V(s,Y z "' s "' x (s)) - V(s,x)\ -» 0, 

hence V{s n , x) — V{s, x) —* as n — > oo. Furthermore, taking any zq € [/, we have from 

^(s, x) - F(s n , x) < x) - [s n - s)h(zo) - EV(s, Y z °' s «' x ), 

which goes to zero as n — > oo by (|13p . and the claim is proved. □ 

Remark 3.5. Notice that, by the Rademacher theorem in infinite dimensions, the previous 
proposition implies that the value function V(s, x) is differentiable in a dense subset of X. 
The local Lipschitz continuity of V also implies that (Clarke's) generalized gradient of V(s, x) 
with respect to x is defined everywhere on X. 

For the following proposition, which establishes a monotonicity property of the value func- 
tion, we need to define the natural ordering in X: we shall write x 1 > x 2 if Xq > Xq and 
x\ > x\ almost everywhere. Similarly, x 1 > x 2 if the previous inequalities hold with the strict 
inequality sign. 

Proposition 3.6. If a\ > and (fo is increasing, then the value function V(s,x) is increasing 
with respect to x in the sense just defined. 

Proof. The proof is completely analogous to that of proposition 14.31 below if we prove that 
A generates a positivity preserving semigroup. This is indeed the case: in fact, a direct 
calculation shows that A is the adjoint of the operator A defined in section [U Since a\ > 0, 
A generates a positivity preserving semigroup S(t). It is well known that A is the generator 
of the adjoint semigroup S(t)*. Let x, y be arbitrary positive elements of X. Then 

0<(S(t)x,y) = (x,S(t)*y). 

By the arbitrariness of x and y, S(t)* is positivity preserving. □ 



3.2 Approximating the value function and the optimal strategy 

Let us now consider the Bellman equation on X associated to the problem of maximizing (JT]), 
which can be written as 

v t + - Tt(GG*v xx ) + {Ax, v x ) + H (v x ) =0, < t < T 
v{T) = Lp, 

where H (p) = sup z£U ({Bz,p) + h(z)). 

The main problem with (I14p is that it is not solvable with any of the techniques currently 
available, with the possible exception of the theory of viscosity solutions. In particular, as of 
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now, one cannot characterize the value function as the (unique) solution, in a suitable sense, of 
equation (|14p . As a consequence, we cannot obtain an optimal strategy for the optimization 
problem at hand. As a (partial) remedy we develop a method to approximate the value 
function and to construct suboptimal feedback strategies that are asymptotically optimal, in 
the sense of Proposition 13.121 below. Let us also briefly recall that, if we know a priori that a 
smooth solution to the Bellman equation exists, then we can apply the verification theorem 
proved in [16] , which in turns allows to obtain precise characterizations of the optimal strategy 
(see subsection 13.31 and section . 

Let us begin proving some approximation results for the value function V. Let e G]0, 1] 
and define G £ : R 2 — > X as 

" (Jo 
eb x 



CL 



Let W\ be a standard real Wiener process independent of Wo, set W = (Wo, W±), and denote 
by F the filtration generated by W. Let IA be the set of F-adapted processes taking values in 
U. 

Consider the following approximating SDE on X: 



dY(t) = [AY(t) + Bz(t)] dt + G £ dW(t), Y(s) = x, 



(15) 



where z €U and < s < t < T. 

For a fixed z, let Y and Y e be, respectively, solutions of ((6]) and of (|15p . Moreover, let us define 

<p e (x) = ((poe(x),0), tpoe = &0e * Ce, <P0e(x) = Po(v)X[~l/e,l/e] and h £ (x) = (-h 0e (x), 0), 

hoe(x) = ho * Cei x )- I n particular ipo £ £ C^(R), ho E G C 2 (M). Finally, the approximate 
objective function and value function are defined as 



ip £2 (Y ei (T)) + / h S2 {z{t))dt 



V £uE2 (s,x) = sup J £u£2 (s,x;z) 



In the following we shall set e = (£1,62) an d lim e _^o := lim £2 >o h m ei-*o- Moreover, for R > 

we set Cr = {x € X : \xq\ < R}. 

Theorem 3.7. One has V £ (s,x) — > V(s,x) as e — > uniformly over s G [0, T\, x G Cr, for 
all R>0. 

Proof. Setting n £l (t) = Y £l (t) — Y(t), one has 

rjeS) = ei f e^^BxdWxir) =: e lV (T), 

J s 

with B\ : R — > X, B\ : x — > (0,b\(-)x), and (suppressing the subscripts on the expectation 
sign for simplicity) 



\J £ (s,x;z)-J(s,x;z)\ < |E[^ 2 (y ei (T)) - <p(Y(T))]\ 

r-T f-T 



+ 



E 



h £2 (z(t))dt 



h(z(t))dt 



< \E[<p ea (V E1 (T)) - <p £2 (Y(T))]\ + |E[^ 2 (F(T)) - <p(Y(T))]\ 



+ 



E 



h £2 {z{t))dt 



h(z(t))dt 



(16) 



Since 



E\Y ei (T) - y (T) I = eiE /"* e {t - r)A B 1 dWi (r) 



0. 
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then Y £l (T) — ► y(T) in probability uniformly over x € X as £1 — > 0, and by the continuous 
mapping theorem ip £2 (Y £l (T)) — ► (p £2 (Y(T)) in probability. Moreover yj £2 (F(r)) — ► <^(y(T)) 
in probability uniformly over x £ Cr, for all R > 0, as e 2 — ► because </?oe 2 (£) ~~ * VoOe) 
dx-a.e. in R. Let us now prove that </? £2 (y ei (T)) is uniformly integrable with respect to e. 
First let us observe, as it is immediate to show, that there exists K, independent of such 
that ip £2 (x) < K{1 + |x|) m . Then we can write 

sup E|^ £2 (y £1 (T))| < sup KE(i + |y(r)+77 £1 (r)|r 

e£]0,l] 2 £i£]0,l] 

< K 1 +K 2 sup (E\Y{T)\ m +e 1 m E|r ? (r)D 

eie]0,l] 

= K 1 +K 2 (E\Y(T)\ m + E\ri(T)\ m ) < 00, (17) 

where we used twice the inequality \x + y\ rn < 2 m (|x| m + \y\ m ) and Burkholder-Davis-Gundy's 
inequality. Furthermore, 

sup E[\<p S2 (Y Sl (T))\x A ] < E[(A^ + K 2 (\Y(T)\ m + HT)\ m ))XA] - (18) 

ee]o,i] 2 

as P(A) -> 0, because #1 + K 2 (\Y(T)\ m + \T](T)\ m has finite expectation. Then CCD and (fTgj) 
imply that y> e2 (Y ei (T)) is uniformly integrable (sec e.g. [22J, lemma 3.10), hence 

|E^ £2 (y £1 (r))-E^y(r))|^o 

as e — ► (see e.g. [22], proposition 3.12). 
Similarly, since 

\he 3 (z(t,u)) - h(z(t,u))\ < Ki + K 2 \z(t,uj)\ m 
for all t € [0,T], u € fi and E |z(*)| w 

< 00 (because U is compact), by the dominated 

convergence theorem we have 

E f \h e2 (z(t)) - h(z(t)\dt -> 
Jo 

as £2 - * 0. 

In view of (|16p we have thus proved that |J £ (s,x;z) — J(s,x;z)| — > uniformly over 
s € [0, T], x G (7r, for all > and 2 £W, hence also that V^s, x) — ► V(s, x). □ 

If the cost function h$ is continuous, one can use a different regularization, without requir- 
ing compactness of U. 

Proposition 3.8. If h is continuous, then the assertion of theorem \3. 7| holds. 
Proof. Let h E s be the sup-inf convolution of h (in the sense of [27]). that is 

I 1 9 1 12 

(\z — y\ \z — x \ 
— o 7^ \-h(y)), 0<5<e. 
2e 2d J 

It is known that h £i $ is differentiable with continuous derivative, that inf^gx h(x) < h £) ${x) < 
h{x) for all x 6 X and that lim £ ( 5_ >0 + h £ ^{x) — ► /i(x) uniformly over x € Cr (see [27]). Setting 
^£2 = h £2>£2 / 2 , proposition 13.11 and the dominated convergence theorem yield 

r-T r-T 

E h E2 (z(t)) dt ^ E h(z(t))dt, 
Jo Jo 

which implies that the third term on the right-hand side of (I16p converges to as e — > 0. □ 
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Theorem 13.71 (or its variant), together with the following result, allow one to approximate 
the value function V(s, x) in terms of the solutions of a sequence of Bellman equations. 



ssume moreover 



Proposition 3.9. Assume that the hypotheses of theorem S. 1 are verified. A 
that ho is strictly convex and tpo is concave. Then the approximate value function V £ is the 
unique mild solution (in the sense of of the Bellman equation 



v t + 7} Tr(G £l G* £l v xx ) + (Ax, v x ) + H Qe2 (v x 

where H 0£2 (p) = swp zeV ((Bz,p) + h £2 (z)). 
Proof. Setting 



v(T) = <p, 



£ 2 J 



B £1 : R — > M , B £ l 



bo/o-o 



the approximating equation (|15|) can be rewritten as 

dY(t) = [AY(t) + G £l B £1 z{t)] dt + G £l dW(t), 



Y(s) 



x. 



(19) 



The state equation (|T9j) . hence also (fT5j) . is covered by the FBSDE approach to semilinear 
PDEs in Hilbert spaces (see e.g. |13j). In order to prove the statement, we shall verify 
that hypothesis 7.1 in [H] holds true. In particular, G £l is Hilbert-Schmidt because b\ G 
L 2 ([— r, 0], R+); (f £2 is Lipschitz because ipo £2 S C^(]R); |i? ei z|x is bounded for z € U because 
b\ G L 2 ([— r, 0], M) and C/ is compact; finally, since ho is proper and positive, it is immediate 
to find £q g]0, 1], C > such that h £2 (x) > —C and inf[/ /i £2 < C, for all positive 82 < £o- 

Since h is strictly convex, for a sufficiently small £2 also /i £2 is strictly convex. Therefore 
we have 

g £2 (p) = argmax((Bz,p) + h £2 (z)) = (h ' )~ 1 (B*p). 



The claim now follows from [14j . theorem 7.2 provided we prove that the closed loop equation 
dY(t) = [AY(t) + Bg(v x (t,Y(t))]dt + G £l dW(t), Y(s) = x. (20) 
admits a solution. In fact this follows as in Theorem 7.2 of [12]. □ 

Remark 3.10. In fact the convexity of U implies that Hq £2 E C 1 (M), and hence that V £ G 
C°^{[0,T],X), as in corollary S21 below. 

The above approximations do not give a way to construct approximately optimal strategies 
for the original problem. In fact, it is well known that the problem of constructing approx- 
imately optimal controls from the knowledge of an approximate value function is very hard, 
and in general unsolved. However, it is possible to construct a (suboptimal) feedback control 
for which we have some error control, in the sense defined below. For a map / : [0, T] x X — > U 
such that the equation 



dY(t) = AY(t)dt + Bf{t,Y{t))dt + GdW{t), Y(s) 



x, 



admits a mild solution Y(t), let us set Uf(t) = f(t,Y(t)) and V^(s,x) = J(s,x;uf). Similarly 
we define V/(s,x). Let us suppose that we can obtain a feedback law / : [0, T] x X — > U, 
which is approximately optimal for the regularized problem, and let us write V £ ~ Vi to mean 
that the two values differ by a small constant. Moreover, recall that V ^ V £ ~ Vi . 

Proposition 3.11. Let f(t,x) be Lipschitz in x uniformly overt. Then v/(s,x) — » V^{s,x) 
as e — > 0. 
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Proof. Denote by and Y £ , respectively, the solutions of the equations 



dY f (t) = AY f (t)dt + Bf(t,Y f (t))dt + GdW(t) 
dY/(t) = AY/{t)dt + Bf(t,Y/(t))dt + G e dW(t), 

with Y f (s) = Y £ f (s) = x. Let us assume, without loss of generality, s = 0. Let us show that 
Yj ' (t) — > Y^(t) in L 1 (J7,P), hence in probability, for all t £ [0, T]: by variation of constants 
we have 

\Y/(t)-Yf(t)\ < f \e^ A B(f( S ,Y/(s))- f(s,Yf(s)))\ds 
Jo 

+e [ e^-^BidW^s) 
Jo 

< / mis )\Y £ f {s)-Y f {s)\ds + e\ / e {t - s)A B 1 dW l {s)l 
Jo Jo 

where m(s) = |e^ t_s ^| \B\ |/|Lip- Taking expectation on both sides and recalling that the 
stochastic convolution has finite mean, Gronwall's lemma yields 



E|y/(t) - Y f (t)\ < eNe^ m{s) 



dn 



(21) 



as e -> 0, hence Y e f (t) -> Y^(t) in probability for all t G [0,T]. 

By the same arguments used in the proof of theorem 13.71 we obtain that 



lim lim E fe (y £ {(T)) = E^(T)). 

£2— >0ei— >0 



(22) 



Similarly, 



\h E2 (f{t,Y £ {{t)))-h(f{t,Yf(t) 



< \h £2 (f(t,Y E {(t)))-h £2 (f(t,Yf(t)))\ 
+ \h e2 (f(t,Yf(t)))-h(f(t,Yf(t)))\ 



and 



h £2 (f(t,Y £ {(t))) ^ h £2 (f(t,Yf(t))) 

in probability as e\ —* for all t G [0, T], because Y £l {t) — > Y?{t) in probability and h £2 o fit, ■) 
is continuous. Furthermore, h E2 (f(t,Y f (t))) -> h(f(t,Y f (t))) P-a.s. for all t G [0,T] as 
£2 — > because h £2 — ► /i a.e. on R. Since |/i £2 (/(i, y £l (i)))| < sup^gjy /i £2 (x) < oo, then 
h £2 (f(t,Yj[(t))) is uniformly integrable with respect to E\, hence 



E 



as S\ — > 0. Finally, 



E 



o 



h £2 {f{t,Y £ {{t)))-h £2 {f(t,Yf{t))) 



h £2 (f(t,Yf(t)))dt-h(f(t,Yf(t))) 



dt^O 



dt^O 



(23) 



(24) 



as £2 — ► by the dominated convergence theorem, taking into account that 

rT 



E 



IM/fo^C*))) -/i(/(t,W(t)))|dt < oo, 



o 



because / is bounded, as follows by the compactness of U. The claim now follows by (|22p . 
(1231) and (EH). □ 
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The previous proposition does not obviously allow one to say that u{t) = f(t,Y(t)) is an 
approximately optimal feedback map for the original problem, as / itself in general depends 
on e±, e 2 . The next proposition gives quantitative estimates on \V^(s,x) — V/(a,x)\. 

Proposition 3.12. Assume that f(t,x) is Lipschitz in x uniformly overt, and that (p, h are 
Lipschitz continuous. Then there exist constants N = iV(|/|Lip) and 5 = 6(82) such that 

\Vf(8,x)-V/(8,x)\<Nei + 6(e 2 ) 

with lim £2 ^o ^(£2) = 0. 
Proof. Let us write 

\vf-v/\ < E|^ 2 (F £ {(r))-^(y/(r))| 



+E 

h + h, 



T 



h £2 (f(t,Y-f(t)))dt-h(f(t,Yf(t))) 



dt 



and 



Then 



h < |E[^ 2 (y £ {(T))-^ 2 (y/(r))]t + |E[^ 2 (y/(T))-^(y/(T))]| 
= hi + hi- 

hi < |^ 2 |LipE|y £ {(T)-y/(T)| < l^lLipeieJ?" 



where we used the fact that mollification does not increase the Lipschitz constant. We also 
have 

J 12 < E[\^ 0e2 (Y f (T))-MY f (T))\;\Y f (T)\<s^ 

+E[|< A)£2 (y ( /(T)) - cp (Y f (Tm \Y f (T)\ > e~ 2 l 
< Si(e 2 ) + o 2 {e 2 ), 



where 



5i(e 2 ) = sup \foe 2 ( x ) ~ fo(x)\ < 00, 

\x\<l/s 2 



as ipo e2 converges to tpo uniformly on compact sets, and #2 (£2) is defined as follows: there exist 
K±, K 2 > such that 



E 



and 



\We 2 (Y f (T)) - MY f (T))\; \Y f (T)\ > e^ 1 ] <E[K 1 + K 2 \Y f (T)\ m ; \Y f (T)\ > ej 1 

y/(T) < e TA x+ [ T e^ A Bf(t,Yf(t))dt+ C e^ A G dW(t) 
Jo Jo 

+ [ T e^ A Rdt+ C e^ A GdW(t) =: fi 2 + Z x . 
Jo Jo 



e TA x 



Similarly, 



Yf{T)>e TA x+ C \^ A rdt+ C e^ A GdW{t) =:^ + Zx, 
Jo Jo 
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where fii, \i 2 G X, U C [r, R], and Z\ is a centered X- valued Gaussian random variable. 
Denoting by Z the K- valued components of Zi, we have 

lo f (T) - Z < (/i 2 ) , Yq(T) - Z > ( m )o, 

hence |y/(T) - Zj < |(/^i) | V |(/i 2 ) o| =: /i, or equivalently I^q < fi + |Zj. In particular, 
Z is a centered Gaussian random variable. Then 



E 



K 1 + K 2 \Y t (T)\ m ;\Y t (T)\ > ^ 1 < /^P(|Z| + /x > ej 1 ) 



+K 2 ¥, 



|Z| + / u) m ;|Z|+ / u> £ 2 1 



Note that 5 2 (e 2 ) -> as e 2 -> since E(|Zj + ^)' m < oo. 
We have 



< 



M/(-,r £1 ))-M/(-,y)) 

-^21 + h2 1 



+ 



^ 2 (/(-,y))-^(/(-,y)) 



Lrp 



where L\ stands for L 1 x [0, T] , dP x dt) . Recalling again that mollification does not increase 
the Lipschitz constant, we also have 

hi < \h\ Up \f\u P \Y ei -Y\ L i < l^lLipl/lupeiiV ( T eti m ^ ds dt. 

Jo 

Finally, using again the uniform convergence on compact sets of mollified continuous functions, 

I 22 <T5 3 (e 2 ). □ 

3.3 An example with explicit solutions 

In this subsection we study in detail the optimal advertising problem with linear reward and 
quadratic cost. In particular, we shall assume h(z) = —(3zq and ^p(x) = 7x0, with /?, 7 > 0. 
In [16] we proved that a solution (in integral sense) of the HJB equation (I14p is of the type 

v{t, x) = (w{t),x) + c(f), t £ [0, T], x G X, 

where w = (wo, wi) : [0, T] — > X and c : [0, T] — > M are given by 

w' Q (t) + a w (t) + / ai(0wi(t,0^ = 0, t€[0,T[ 

J —r 

w (T) = 7, 

m(t,0 = w (t - Qx[0,T](t ~ £)> 



(25) 



c(t) 



4/? 



ds, 



t€[0,T\. 



Moreover, the optimal strategy is 

(B,v x (t))+ (B,w(t)Y 



z*{t) 



2/3 



2(3 



te[0,T] 



(26) 



(see [16] for more details). 

We extend now the analysis of this specific situation. Let us begin with a rather explicit 
characterization of the optimal trajectory, which could be numerically approximated simply 
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by solving a linear ODE with delay. In particular, let w = (wq,wi) be the solution of (|25l) . 
Then, setting z*(t) = 1 g(B,w(t)) + , the optimal trajectory is the R-valued component Yq of 
the (mild) solution of the abstract SDE 

dY(t) = [AY(t) + Bz* (t)] dt + G dW{t) , 

which is given by 

Y(t) = e tA Y(0) + f e {t - s ^ A Bz*{s)ds+ f e {t ~ s)A G dW{s). 
Jo Jo 

In particular Y is a X-valued Gaussian process with mean and covariance operator 

Mt = e ' A Y(0) + /"' e( J -^B/( S ) ds, Q t = f e ^ A GG*e^ A * ds, 
Jo Jo 

respectively. It follows that Yq is a Gaussian process itself with mean 

EY (t) = (iM, ei ) x = (e tA Y(0),e l ) x + ^j\^ A Bz*(s)ds,e 1 



x 



Y(0),e tA * ei ) + f (Bz*(s),e^ A * ei ) ds, (27) 
/ x Jq \ ix 

where &\ = (1,0) G X. 

Since Y(0) is given as in Proposition 12. II and Bz*(-) is also easy to compute (z* is one dimen- 
sional and B is just multiplication by a fixed vector in X), we are left with the problem of 
computing e tA * e\. However, as one can prove by a direct calculation, the semigroup e tA * is 
given by 

e tA *(x , Xl (-)) = (<Ht),# + 6b[-r,0]^ 
where 4>{-) solves the linear ODE with delay 



a <A(t) + y ai(£)<K* + £)d£, 0<t<T 



d<j){t) 

dt ~ u ' rw ' 1 /_ r ~ J - v1 "' rv " ' * y ~*' (28) 
<A(0) = x ; 0(6=^i(O V£e[-r,0]. 

Therefore e tyl *ei is given by {(f>(t), 4>(t + £)|g g [_ rj o]), where 4> solves ([281) with initial condition 
xq = 1, xi(-) = 0. Such (p can be computed numerically by discretizing (|28|) . and then EYo(^) 
can be obtained by approximating the integrals in (j27[) with finite sums. 

Analogously one can write the variance of the optimal trajectory in such a way that it can 
be easily approximated by numerical methods. In particular, one has 



VarY (t) = (Q t ei, ei) = I J e ^ A GG*e^ A * dse u ex 
= £(e^ A GG*e^ A *e l ,e l )ds 
= J*(Ge^ A *e 1 ,Ge^ A *e 1 )ds 

= f Ge^-^a 2 ds. (29) 
Jo 

Setting i/j(s) = e^~ s ^ A * e\, which can be approximated as indicated before, one finally has 

VarFo(t) = o 2 [ ijj(s) 2 ds. 
Jo 
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One can also perform simple comparative statics on the value function. For instance we 
can compute explicitly its sensitivity with respect to the (maximal) delay r: 

dV dc 

— (t,x;r) = —{wi(t),x 1 ) L 2 (hrM) + —(t;r) 

= Wl (t,-r) Xl {-r) + ^ J\b,w{s))^. (J° h(€)wi(8,0dt\ ds 

bA-r) f T 

= wi(t, -r)xi(-r) H ^— J (B,w(s))wi(s,-r)ds, 

where we have used the fact that (B,w(t)) + = (B,w(t)). 

Note that in the above expression everything can be computed explicitly, as soon as we fix the 
delay kernel b±. Let us consider, as an example, the special case of 61(C) = ^iX[-r,o](0> where 
on the right-hand side, with a slight abuse of notation, b\ is a positive constant. One has 

—(t,x;r) = wi(t, -r)xi(-r) + —j wi(s, -r) (b w Q (s) + h J W\ (s, g) d£) ds. 

Furthermore, if we consider the special case of delay in the control only, that is a±(-) = 0, we 
obtain, after some calculations, 

^(t,x;r) = 7e »o(T-^) Xi( _ r) _ J^-W&o - -(1 - e^))(l - e^ T ~\ 
or 4pao V ao ' 

for t € [r, T] . 

In the special case of oi(-) = an explicit solution of (|25p is easily obtained. This solvability 
in closed form then "propagates" to other quantities of interest. In fact, note that (|25l) reduces 
to 

f w' Q (t) + a w (t) = 

(30) 

t w {T) = 7, 

yielding 
and therefore 

Mt,0 =7e (T - (t+ ^°X[o,T](t + c(t) = j t T {B,W ^ )? ds. 

That is, the last three formulae explicitly give a solution of the HJB equation (|14|) in our 
specific case. 

As a consequence we can also determine the unique optimal feedback control z* as follows: 



(B,v x }+ _ (B,w(t))+ 7 e (T - f)Q ° 
Z { ) ~ 2(3 ~ 2/3 "2/3 



bo+ f bAOe-^x^t + Odt 

J —r 



(31) 



The optimal trajectory can be characterized in a completely similar way as above, with 
the difference that now we can explicitly write: 

e M *ei = (e aot ,e^| 5e[ „ r>0] ), 

hence simplifying (|27p in the present case. Even simpler is the expression for the variance of 
the optimal trajectory, which can be obtained by (j29|) : 

„2 

)■ 



ft 2 

V&rY (t) = a 2 / e 2a °(*- s) ds = (e 2a °* - 1 

Jo 2 «o 
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Optimal advertising policy z*{t) 




2 4 6 8 10 

Time t 

Figure 1: Optimal advertising policy in four different "churn" settings 

Sharp characterizations of the optimal advertising trajectory as well as the resulting ex- 
pected profit functions in the case of linear reward and quadratic cost function allow for 
interesting observations regarding the importance of the proper accounting for the memory 
effects in planning the advertising campaign. Figure [T] displays the optimal advertising spend- 
ing rates z*(t), as expressed by (j26]) . with oq(£) = aie~^^ Sa and &i(£) = bie~^" Sb in four 
different settings: a% = b\ = ("no churn"), a\ = —5, b\ = ("goodwill churn"), a\ = 0, 
b\ = 5 ("advertising churn"), a\ = —5, b\ = 5 ("goodwill-advertising churn"). Note that in 
the absence of churn, the optimal advertising trajectory, as implied by (|3ip . is a monotone 
function of time with z*(t) = 760/(2/?). While the advertising rates are similar in all settings 
in the beginning of the pre-launch period as well as right before the product launch time T, 
the details of advertising policies differ dramatically in the middle of the pre-launch period. 
For example, in the presence of a strong "goodwill churn" the optimal advertising trajectory 
takes a characteristic "impulse" shape, while in the strong "advertising churn" setting the 
optimal advertising spending quickly builds up a strong goodwill level in the middle of the 
pre-launch region, slowing down significantly right before the product launch. When the pres- 
ence of both types of "churn" is pronounced, the optimal advertising policy is represented by 
a set of advertising sprees with rapidly growing intensity. Figure [2] illustrates how the strong 
influence of memory effects on the shape of optimal advertising policies translates into perfor- 
mance differences between the optimal advertising policies and the policies which neglect the 
presence of advertising delays. In this figure we plot the relative difference 

V(0,x)-V°(0,x) 
V(0,x) 
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a) b) 




Amplitude of the goodwill delay a x Amplitude of the advertising delay 



Figure 2: Relative performance gap of the "memoryless" advertising policy 

between the optimal expected profit function V(0,x) and the expected profit value V°(0,x) 
obtained by applying, for t G [0, T], the advertising policy z°(t) = 7&oe'- T_ ^ ao /(2/3) optimal 
in the absence of memory effects (i.e. in the setting a\ = b\ = 0). This relative difference is 
plotted as a function of the amplitude of the "goodwill churn" term a\ (Figure Eh.) , and as a 
function of the amplitude of the "advertising churn" term b\ (Figure [2b). The initial goodwill 
conditions were selected as xq = 10 and X\(£) = xoe~'^' for £ G [— r, 0] and the advertising 
history 2o(£) was set equal to for £ G [— r, 0]. We observe that the relative loss of efficiency 
associated with the use of the "memoryless" policy z°(t) can be quite significant - in the 
examples we use it exceeds 5% and can be as high as 20% in settings with strong "churn" 
effects. 

4 The case of delay in the state term only 

In this section we consider a model for the dynamics of goodwill with forgetting, but without 
lags in the effect of advertising expenditure (carryover), i.e. with &i(-) = in ([T]). An analysis 
of this model was sketched in |16j . where only an abstract existence result was given. Here we 
present a more refined result (see theorem 14.51 below) and obtain some qualitative properties 
of the value function, together with a characterization of the optimal strategies in terms of the 
value function in two specific cases. In particular, theorem 14.51 formulates sufficient conditions 
ensuring that the value function is the unique solution (in a suitable sense) of the associated 
Bellman equation and that the optimal advertising policy is of the feedback type. Let us 
recall once again that such a situation is not possible in the more general case discussed in 
the previous section. Moreover, in the case of linear cost function, the optimal control takes 
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a particularly simple "bang-bang" form (Corollary 14. 10p . 

Let us also mention that for stochastic control problem with delay terms in the state 
variable one can apply both the approach of Hamilton- Jacobi equations in L 2 spaces developed 
by Goldys and Gozzi [pS], and the forward-backward SDE approach of Fuhrman and Tessitore 
[12j . We follow here the first approach, showing that both the value function and the optimal 
advertising policy can be characterized in terms of the solution of a Bellman equation in 
infinite dimensions. 

We assume, for the sake of simplicity, that the goodwill evolves according to the following 
equation, where the distribution of the forgetting factor is concentrated on a point: 

dy(t) = [a y(t) + aiy(t - r) + b z(t)] dt + a dW (t), < s < t < T 

(32) 

y(s)=x ; i/(s + f)=zi(0> £e[-r,0]. 

The following standard infinite dimensional Markovian reformulation of this dynamics will 
turn out to be useful. 

Let us define the operator A : D(A) C X — > X as 

A:(x , Xl (-)) ^ (aoxo + aizi (-rJ.zU-)), D(A) = lx W 1,2 ([—r, 0];R). 

It is well-known (see e.g. [8]) that A is the generator of a strongly continuous semigroup S(t) 
on X. More precisely, one has 

S(t)(x ,Xi) = (u(t),u(t + Q\z e [- r ,0]), 

where u(-) is the solution of the deterministic delay equation 

= a u(t) +aiu(t - r), 0<t<T 
dt (33) 

u(0) =x ; u(0=xi(Z), ee [-r,0]. 

Furthermore, set z = (a~ 1 boz, ^i(-)), with z\{-) a fictitious control taking values in L 2 ([— r, 0],R) 
and define G : X — > X as 

G : (xq,xx(-)) h-> (ax ,0). 

Let W\ be a cylindrical Wiener process taking values in L 2 ([—r, 0],R), so that W = (W , Wi) 
is an X-valued cylindrical Wiener process. 

Chojnowska-Michalik [6] proved the following equivalence result. 

Lemma 4.1. LetY = (Yq,Y{) be the unique mild solution of the following stochastic evolution 
equation on X : 

dY(t) = (AY(t) + Gz{t)) dt + GdW(t), < s < t < T, 

(34) 

Y(s) = x. 

Then Yo(t) solves the stochastic delay equation $32\). 
Define h : X -> R and tp : X -> R as 

h(x ,xi) = -h (abQ 1 x ) 
ip(x ,xi) = ipo(x ). 

Then we have, thanks to lemma 



J(s, x; z) = E 



s,x 



V{Y{T)) + j\{z{s)) 



ds 
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and 



V(s, x) = sup J(s, x; z), (35) 



where Z denotes the set of all strategies z : [0, T] — > U x L 2 ([—r, 0],R) adapted to the filtration 
generated by 1", and U is the image of U under the action of the map x t— > a~ 1 box. 

We can now prove some qualitative properties of the value function. 

Proposition 4.2. Ifipo is concave and ho is convex, then the value function V(s,x) is concave 
with respect to x. 

Proof. Identical to the proof of proposition 13.21 thus omitted. □ 

In the following proposition we use the ordering in X defined right before Proposition 13.61 

Proposition 4.3. Let a\ > and (po be increasing. Then the value function V(s,x) is 
increasing with respect to x. Moreover, if (po is strictly increasing, then the value function 
V(s,x) is strictly increasing with respect to x, and V{s,x l ) = V(s,x 2 ) if and only if x 1 = x 2 . 

Proof. Let x 1 > x 2 in the sense just defined. One has 

J(s,x 1 ;z)-J(s,x 2 ;z) = E^tp(y s ' xl ' z (T)) — (p(y s ' x2,z (T)) 

= E U^"^ 1 + C) - v(e A{T ~ s) x 2 + C) 

where ( := /J e A{ - T ~^Gz(t) dt + jj e A{ - T ~^G dW{t). The assumption oi > together with 
03*]) implies that the semigroup generated by A is positivity preserving, i.e. x 1 > x 2 implies 
e A(T—s) x i y e A(T-s) x 2 _ Therefore, by the monotonicity of ipo, one also has (p(e A ^ r ~ s ' x 1 + C) > 
i^^ e - A { T - s )x 2 +C,) a.s., hence J(s, x 1 ; z) > J(s, x 2 ; z), and finally V(s, x 1 ) = sup zg ^ J(s, x 1 ; z) > 
sup 2g ^ J(s,x 2 ;z) = V(s,x 2 ). The other assertions follow analogously, using 03]) . □ 

Remark 4.4. In the above proof the positivity preserving property of the semigroup e tA is 
crucial, and the assumption a\ > is "sharp" in the following sense: if a\ < 0, one can find 
x > such that e tA inverts the sign, i.e. e tA x < 0. 

Moreover, under the assumptions of the theorem, the value function is increasing with 
respect to the real valued component of the initial datum. By this we mean that given 
x 1 > x 2 with Xq > Xq and e., then V(s,x 1 ) > V(s,x 2 ). Therefore one also has 

DqV > 0. The sub differential can be replaced by the derivative if we can guarantee that V is 
continuously differentiable with respect to Xq. Conditions for the continuous differentiability 
of V with respect to x are given in proposition 14.61 below. 

In contrast with the general case considered in the previous section, if delay enters only the 
state term, then it is possible to uniquely solve the associated Bellman equation, and thus to 
characterize the value function and construct optimal strategies. The following result, which 
relies on |Tj3] , gives precise conditions for the above assertions to hold. 

Theorem 4.5. Assume that ho is convex, let Hq : R 9 p i— ► sup ueU (pbou — ho(u)), and suppose 
that ao < — a\ < a/7 2 + a^, where 700th 7 = ao, 7 g]0, 7r[. Then the value function V(s,x) 
coincides [x-a.e. with the mild solution in L 2 (X,fi) (in the sense of 115^ ) of the equation 



1 Z" 
d t v + -o- 2 d 2 v + (a x + a lX i(-r))d v + J x'^dMO # + H (d v) = 

v{T,x ,xi) = ip (x ), 



(36) 
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where /i is a measure of full support on X. Moreover, the optimal strategy admits the feedback 
representation 

z*(t) G ab^D^H{ad v{t,Y^{t),Y^{t)), (37) 

with H defined in (Jty , provided there exists a solution Y^t), Y*(t) of the closed-loop differ- 
ential inclusion 

{ dY (t) G [a Y (t) + a 1 Y (t-r) + aD^H(ad v(t,Y *(t),Y 1 *(t))]dt + adW (t) 

d ( 38 ) 
dY l (t)(0 = ^Y 1 (t)(0. 

Proof. By the usual heuristic application of the dynamic programming principle one can as- 
sociate to the control problem (|35jl the following Hamilton- Jacobi-Bellman equation on X: 



' v t + - Tr(GG*v xx ) + {Ax, v x ) + H {v x ) = 0, < t < T, 

1 ( 39 ) 
, v(T,x) = ip(x), 

which coincides, after some calculations, with (|36jl . Note that the Hamiltonian Hq can be 
regarded as a function of X in K and can be equivalently written as 



H Q (p) = H (p ) = sup ( (Gz,p) + h(z) 

z£UxL 2 ([-r,0],B 



= sup (azopo- ho{o-b 1 z )j. (40) 

In order to apply the results of [15 j . we also need to define 

H(q) = H(q ) = H (a~ 1 q ) = sup (zoq - ho^ab^zoj) ■ (41) 

z &U 

Since Hq is bounded from below, (|41|) implies that H is Lipschitz continuous (in R and in X). 
Moreover, the assumption on oo, a\ and assumption (i) of section [2] imply that the uncontrolled 
version of (f3~4"|) . i.e. 

dY{t) = AY{t)dt + GdW{t), (42) 

admits a unique non-degenerate invariant measure fi on X (see [8]), which is Gaussian with 
mean zero and covariance operator Qoo = Jo°° e sA GG*e s A ds. In particular, the restriction of 
fi on the M-valued component of X has a density p(x) = e ~\ x \ 2 / 2v ' 2 for some v > 0. This 
implies that ip S L 2 (X,/i): in fact, 



\ip(x)\ 2 n(dx) 

x 



[ \ip (x)\ 2 p(x) dx < K f (1 + \x\) m e-\ x \ 2/2u2 dx < oo. 



Therefore, theorems 3.7 and 5.7 of [15 J yield the existence and uniqueness of a solution in 
L 2 (X,n) of (|39|) . or equivalently of (|36|) . which coincides /i-a.e. with the value function V. 
Finally, observing that the maximum in (|4ip is reached by D^H^o) (setting, if needed, 
ho(x) = +oo for x £ U), a slight modification of the proof of theorem 5.7 in [15] shows that 
the optimal strategy is given by zfi(t) G Dq H(ad v(t, Y£(t), Y{(t)), where Y* = (Y£,Y{) is 
a solution (if any) of the stochastic differential inclusion (|38p . The relation z*(t) = ab^ 1 ^^) 
thus completes the proof. □ 



Let us briefly comment on the previous result: the HJB equation (|36|) is "genuinely" 
infinite dimensional, i.e. it reduces to a finite dimensional one only in very special cases. 
For example, by the results in [26], (I36p reduces to a finite dimensional PDE if and only if 
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ao = — a\. However, under this assumption, we cannot guarantee the existence of a non- 
degenerate invariant measure for the Ornstein-Uhlenbeck semigroup associated to (|42|) . Even 
more extreme would be the situation of distributed forgetting time: in this case the HJB 
equation is finite dimensional only if the term accounting for distributed forgetting vanishes 
altogether. Moreover, note that if a\ is negative, i.e. it can be interpreted as a deterioration 
factor, the assumption of the theorem says that a\ cannot be "much more negative" than ao- 
On the other hand, if a\ is positive, then the improvement effect as measured by a\ cannot 
exceed the deterioration effect as measured by |eto|- In essence, the condition on ao, ai, which 
is needed to ensure existence of an invariant measure for equation (|42p . does not impose severe 
restrictions on the dynamics of goodwill. 

If the data of the problem are smoother, a different approach allows one to obtain regularity 
of the value function. 

Proposition 4.6. Assume that ip G C 1 ^), \^ (x)\ < K(l + \x\) m , and H G C l {R). Then 
V G C°^([0,T] x X). 

Proof. Follows by the regularity results for solutions of semilinear partial differential equations 
in Hilbert spaces obtained through the FBSDE approach. In particular, denoting by C a 
positive constant, boundedness of U implies that l&o-^l < C, \ho(z)\ < K(l + \z\) rn < C, and 
finally Hq is Lipschitz as follows by 

\H (p) - H (q)\ = \sup{(Bp,z) - h (z)) - sup((Bp, z) - h (z))\ 

z&U z£U 

< snp\((p-q),B*z)\<C\p-q\. 

z€U 

Then all hypotheses of |13j . theorem 4.3.1, are satisfied, which yields the claim. □ 

Corollary 4.7. Let ipo be as in proposition ^. 6\ and ho strictly convex. Then V G C 0,1 ([0, T] x 
X). 

Proof. Since convexity implies continuity in the interior of the domain, then ho(U) is bounded. 
Extending ho as ho(x) = +oo for x ^ U, ho is clearly 1-coercive, hence H is convex and finite 
on the whole R ([21j. prop. E.l.3.8). The strict convexity of ho implies that Ho is continuously 
differentiable in the interior of its domain, i.e. on R ([21], thm. E.4.1.1). Then the smoothness 
of V follows again by [13], theorem 4.3.1. □ 



Remark 4.8. We should also mention that in the framework of the FBSDE approach to HJB 
equations ([13]), if the Hamiltonian H and the terminal condition ip satisfy some smoothness 
and boundedness conditions, then we do not need the assumption about the existence of an 
invariant measure for the uncontrolled state equation. The approach used above ([E]), while 
requiring the existence of the above mentioned invariant measure, allows for more singular 
data (for instance one could choose <po(x) = —M, M » 0, for x G M_, and (fo(x) > for 
x G R+). 



In general, obtaining explicit expressions of the value function V trying to solve (|36[) 
is impossible. However, under specific assumptions on the model we can obtain stronger 
characterizations, at least from a qualitative point of view, of the value function and/or of the 
optimal strategy. 

Corollary 4.9. Assume that ho(x) = f3x 2 and U = [0,R], R < oo. Then the optimal strategy 
is given by 

( 0, D V* < 



z*(t) 



h -^-, 0<D V*<2b 1 m (43) 
R, D V* > 2b ~ 1 pR, 
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where V* := V(t,Y *(t),Y{(t)). 
Proof. One has 



H (p) = sup ( (Gp, z) + h(z)\ = sup (ap z - (3z ) 

0<z o <R 0<2 <.R 



2 , 



where R = a 1 boR and /3 := <r 2 fr 2 /5- Therefore 

£T( g ) = 9o/4/3/ {0 < go < 2 ^ } + (qoR - PR 2 ) \ qo>2 pR} 

and 

DH(q) = 9o/2/3/ {0 < (?0 < 2( g / j } + RI {qo>2 pR}- 
Theorem 14.51 now yields (03]). □ 

Note that whenever y?o is increasing, we get DqV* > 0, hence the optimal control is either 
linear in DqV* or constant for DqV* over a threshold. 

Corollary 4.10. Assume that ho(x) = fix. Then the optimal strategy is of the bang-bang type 
and is given by 

' 0, D V* < abo 2 /3 
p, D Q V* = abffi 
t R, D Q V* > abfp, 



Z*(t) : 

where p is an arbitrary real number. 



Proof. Setting R = a 1 bQR and (3 := cr 2 b (3, one has 

H {p) = sup (ap z - (3z ) = (ap - /3)RI {po> p /a} 
o<z <ii 

and H(q) = (q - (3)RI {qo> ^ } , thus 

D-H(q) = RL sx + RIj 

Theorem 14.51 yields the conclusion. □ 

In general, even specifying a functional form of ho, an explicit solution of the HJB equation 
for arbitrary (po is not available, hence the above expressions of the optimal control strategy 
in terms of the value function and their corresponding qualitative properties are the "best" 
that one can expect, at least in the cases we have considered. 



5 Concluding remarks 

A number of deterministic advertising models allowing for delay effects have been proposed 
in the literature. However, the corresponding problems in the stochastic setting have not 
been investigated. One of the reasons is certainly that a theory of continuous-time stochastic 
control with delays has only been developed recently, following two approaches. The first ap- 
proach is based on the solution of an associated infinite-dimensional Hamilton- Jacobi-Bellman 
equation in spaces of integrable functions (see [15] ) . The other one relies on the analysis of an 
appropriate infinite-dimensional forward-backward stochastic differential equation (see [T2]). 
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Both approaches, however, cannot be applied to problems with distributed lag in the effect of 
advertising. 

Problems with memory effects in both the state and the control have been studied first 
by Vinter and Kwong (in a deterministic LQ setting), and by Gozzi and Marinelli [TB] 
(in the case of linear stochastic dynamics and general objective function). A general theory 
of solvability of corresponding HJB equations is currently not available, while an infinite- 
dimensional Markovian reformulation and a "smooth" verification theorem have been proved 
in pi]. 

In this paper we have concentrated on deriving qualitative properties of the value function 
(such as convexity, monotonicity with respect to initial conditions, smoothness). For specific 
choices of the reward and cost functions, we obtain more explicit characterizations of value 
function and optimal state-control pair. 

While our work makes a substantial initial step in the analysis of the stochastic advertising 
problems with delays, more remains to be done. Potential extensions of the present work 
include the analysis of problems with budget constraints as well as problems of advertising 
through multiple media outlets with different delay characteristics. 
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